Using course-subject Co-occurrence (CSCO) to reveal the structure of an academic discipline: A framework to evaluate different inputs of a domain map

This article proposes, exemplifies, and validates the use of course-subject co-occurrence (CSCO) data to generate topic maps of an academic discipline. A CSCO event is when two course-subjects are taught in the same academic year by the same teacher. 61,856 CSCO events were extracted from the 2010-11 directory of the American Association of Law Schools and used to visualize the structure of law school education in the United States. Different normalization, ordination (layout), and clustering algorithms were compared and the best performing algorithm of each type was used to generate the final map. Validation studies demonstrate that CSCO produces topic maps that are consistent with expert opinion and four other indicators of the topical similarity of law school course-subjects. This research is the first to use CSCO to produce a visualization of a domain. It is also the first to use an expanded, multipart gold-standard to evaluate the validity of domain maps and the intermediate steps in their creation. It is suggested that the framework used herein may be adopted for other studies that compare different inputs of a domain map in order to empirically derive the best maps as measured against extrinsic sources of topical similarity (gold standards). Introduction This article seeks to ascertain the similarity of legal course-subjects in terms of their topical relatedness and to rigorously and in a replicable manner, best distribute those course-subjects in a two-dimensional mapping so that they may be quickly perceived by the viewer using the distance-similarity metaphor (Montello et al., 2003). Once created, domain maps provide cognitive scaffolding for learning (Greenfield, 1984; Wood et al., 1976). These big-picture, global perspectives have the potential to allow a novice to more quickly become familiar with the domain and experts to contextualize their teaching and research in a broader perspective. Additionally, domain maps of legal course-subjects allow for numerous thematic overlays that facilitate insight about legal education in the United States. While it would be possible to create a domain map from the citation interlinkages or lexical overlaps of the scholarly literature, the maps created herein most directly represent how topics are taught and interrelate in almost 200 (ABA & LSAC, 2012) law schools in the United States. The framework used herein may be adopted by other researchers for studies that compare different inputs of a domain map in order to empirically derive the best maps as measured against extrinsic sources of topical similarity (gold standards). This is a preprint of an article published as: Hook, P. A. (2017). Using Course-Subject Co-Occurrence (CSCO) to Reveal the Structure of an Academic Discipline: A Framework to Evaluate Different Inputs of a Domain Map. Journal of the Association for Information Science and Technology, 68(1), 182-196. Please see: http://onlinelibrary.wiley.com/doi/10.1002/asi.23630/abstract. Note: this preprint has been updated to reflect changes in the final version. Cite as: Hook, P. A. (2017). Using Course-Subject Co-Occurrence (CSCO) to Reveal the Structure of an Academic Discipline: A Framework to Evaluate Different Inputs of a Domain Map. Journal of the Association for Information Science and Technology, 68(1), 182-196. The explanatory power of CSCO networks is premised on the assumption that in the aggregate, and for reasons of efficiency, faculty members specialize and focus their energy teaching courses that are topically similar to other courses they teach. The first research question is whether course-subject co-occurrence (CSCO) can be used to produce topic maps that are consistent with expert opinion and other indicators of the topical similarity of law school course-subjects. The second question is, when using CSCO network data to compare normalization algorithms, spatial layout techniques, and clustering algorithms, which combination of algorithms, tools, and techniques is best at portraying the overall structure of law school course-subjects as compared to an extrinsic ‘gold-standard’ of similar course-subject pairs. Related Work While the course-subject structure of legal academia in the United States has been described in essays (Kennedy, 1983) and other writings on the history of law school education (Stevens, 1983), it has never been revealed through the exploration of large datasets and determined through replicable, empirical means. The most similar studies to this work also involve a spatial mapping analysis of either academic courses or disciplines (Biglan, 1973; White & Calhoun, 1984; White & Nolt, 1987). There have been some studies that compare different domain map production techniques (Boyack et al., 2005; Klavans & Boyack, 2006; Van Eck & Waltman, 2009). There needs to be significantly more of these in order to arrive at a consensus for the best combinations of the constituent domain mapping techniques for specific purposes. The use of CSCO networks to make structural claims about a domain is supported by the numerous uses of co-occurrence data that have been used to create domain maps. The underlying assumption is that items that co-occur together are categorically or substantively more similar than those that do not. This includes co-voting (judicial: (Hook, 2007a, 2014a; Pritchett, 1941, 1942, 1948, 1954; Schubert, 1962, 1963; Sirovich, 2003; Thurstone & Degan, 1951)) (legislative: (Clinton et al., 2004; Clinton & Meirowitz, 2001; Jackman, 2001; Moody & Mucha, 2013; Poole & Rosenthal, 1985)), word co-occurrence (Doyle, 1961, 1962), bibliometric coupling (Kessler, 1963; Price & Schiminovich, 1968), co-authorship (De Solla Price & Beaver, 1966), co-citation (Marshakova, 1973; Small, 1973), conomination (Lenk, 1983), co-courses taken (White & Calhoun, 1984; White & Nolt, 1987), coclassification (Hook, 2007b; Spasser, 1997; Todorov, 1989), and co-membership (McCain, 1993). Data and Evaluative Gold-Standard Course-subject co-occurrence (CSCO), is the same professor teaching multiple, different coursesubjects over some period of time. Used herein, the period of time is one academic year as captured in the annual directories of the American Association of Law Schools (AALS). Furthermore, courses with differing individual course names are controlled through a proscribed subject vocabulary supplied by the AALS. In other words, courses with similar content, but with differing titles, are harmonized through a common course-subject listing. In 2010-11, there were 104 academic course-subjects (AALS, 2010). If a professor teaches two different course-subjects in a given year, those course-subjects are connected by a single link when the two mode network (professors and course-subjects) is collapsed to a single mode network (course-subjects). When two professors teach the same two course-subjects, this results in an edge weight between those course-subjects of two when the network is collapsed from a two mode network to a single mode network. In 2010-11, 536 faculty members taught both Criminal Law and Criminal Procedure—the highest amount of pairwise co-occurrence between any of the 104 coursesubjects. At the other end of the spectrum, 1,467 of the 5,356 possible course-subject pairs (((104 x 104) 104) / 2) were not taught by any of the same faculty members. Table 1. Distribution of the Amount of Course-Subjects Taught in 2010-11.